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ABSTRACT * . ♦ * 

In this paper* the steps implemented' by the National 
Institutes of Health (NIH) anfl the National Science Foundation (RSF) 
to upgrade the quality of thef 19T3 Graduate Science Student Support 
Survey are cited. Changes which vere incorporated . into the survey 
were intended to increase the! scope and ^accuracy of data collected, ✓ 
To verify "the coverage, fi liatl of the institutions that v made up th# 
population surveyed in 1973 Wis compared with, lists of Ph .D. -granting 
institutions maintained % by fa<^eral agencies and professional' 
associations with an interest in* highe^ education^ DiscripaDcies were 
checked" by telephone inquiry' and adjustments in the^survey base were 
m^de. The second major component of* the study was to -reconstruct the 
# 1973 survey responses from departmental records for a , sample' of 
departments ai)4 to compare the resulting data with ,that which' had 
been reported fy the departaent^. Al£o 1# selected student interviews 
were held insider to make comparisons^with departmental records. 
1Author/CP) 
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INTRODUCTION _ 

•I would like to review with you today the steps imple*- 
mented by *NSF and NIH in the 1973 GSSS Survey, for the purpose "of 
Upgrading the' quality of this study- ' I "will* discuss one of these 
steps, the 'quality assessment study, in* some depth, giving 
methodology employed ,* findings ,^ and resulting recommendations. * 

Xhe 1373 survey was designed' by NSF and NIH to include 
an examination of the coverage of institutions and. departments to 
assess its adequacy *and to develop a tool f or v expanding coverage 
to encompass the entire population-. The work accomplished dujcing" 
the 1973 survey year also included 9 the development of a -Quick 
Response Sutvey, which was designed tp collect a limited set'Qf 
graduate student enrollment character i'stics frprrv a sample of' depart 
ments-, use this data for pro jectihg-'ghanges in. national graduate 
"enrollment f or 'r^& jor areas of science and engineering, and rn^ke 
the information on\ these changes available to iisefrs several 'mont>hs 
prior to publication of tlxe final purvey statistics,* * '■ 

NS? and NIH. also formed an advisory panel made u^p*of 
both, suppliers and use2?a of .the GSSS Survey data/ This 'advisory • 
paftej. was convened twice to review progression the*1973 survey a*n0 
give us -the benefit of "their thoughts regarding the existing survey 
methods^, instruments, analysis and publication plans. 

•In addition, N&EVand NIH provided for the collection*' \ ' 
of data to he used in the assessment of the quality of ^.he survey 
"data. This quali/ty assessment study. had. two major components, ~ 
The f ir»fet involved ^the collection of enrollment and studerrt 
chara6teristic, data from & sample of departments whicFT wd? # * to be 
compared with that .reporte^i by'the departments ift theiif 1973 survey 
response/ The second component involved a bri^f ^telephone inter- 
view with'* a- saipp^e *6f graduate , students and postdoc^orals so' £hat 
their ' enrollment characteristics and sources of „supfcort cou}.d, fre 
'determined. This data was compared with that 6btained from % 



depaftment records for the. same students. This provided some in- 
sight's into the consistency of the 'students 1 perceptions and the 
department records. 




* 

I would like to discuss with yQii the methodology employed 

in each^ of these study components, and to present the principal 
£ indir^s . , 
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There Vere two aspects to the investigation of -coverage. 
These dealt with coverage of institutions and the adequacy pf 
coverage -of departments within the institutions, s 

. 4 As you probably know, the GSSS Survey population is 
comprised of all institutions., both public and . private v th^t have^ 
at least one Ph.D. program in science or engineering. AIL science 
and engineering, departments within an eligible institution^ that 
have graduate programs offering -a masters or doctorate degrefe are 
asked to resoond to the survey.* A list of 1 the institutions that 
made up the population surveyed in 1973 was compared with lists 
of Plvb. granting institutions niaintained^by Federal agencies and. 
professional associations with an intei^st in higher education. ^ 
Institutions which did rfiot appear in the 1973 GSSS survey 
population, but .which appeared to be eligible for inclusion, were 
contacted by telephone. If it' was established' that an institution 
was eligible, 'we obtained a copy of their graduate school catalog. 
These -catalogs' were then reviewed, and a list of the departments . 
•offering graduate programs was. compiled*. These lists were then ' 
forwarded to th£ graduate deans with a request that they review 
'and modify the list to include all departments that. had 'graduate 
programs-. '; By vising .^hes^" procedures , 14 institutions that were < 
eligible for' inclusion in the survey .population , but which' ha'd not 
been, included*, were identified. The 14 institutions contained a 
total of 102 departments- with gr^dua,te programs, with the - depart- 
ments^ representing less than 2 percent of the number that had 
responded to .the 1973 survey.- 



, The investigation of* the coverage of departments was 
V . * ' *■ , 

conducted by developing a "universe of eligible departments", , • 

which was accomplished "by using the 197 3 survey response and the 
graduate school catalog for each respondeat institution. The "■ 
catalog review revealed, together with the universe of departments 
in. medical schools furnished by NIH, 7,876 science and engineering 
departments. Responses were received from' 6,-559 departments on' 
the 1973' GSSS Survey. The difference of 1,317 departments is 
approximately 20 percent of those responding to \the survey. To 
estimated the Aumber pi full-time graduate students who are 
en-rolled in these 1,317 departments,' average enrollment was deter- 
mined for 52 areas of science' and engineering departments. Apply- 
ing these averages to the 1,317 "eligible bu«t missing 11 departments 
one obtains an estimate of undercoverage of/ .30, 072 full-time 
gr&dua.te students. This represents 18.3 perbent of the 164,318 

^counted in- the 1973 survey-. These estimates of departmental and 
student 'undercoverage 'are considered possibly to be seriously in- 
flated. It is felt that c«ttalbgs tend to overstate s program offer- 
ings, and a sound ^argument, can b6 made that the "eligible but < * 
missing" 'departments would tend to have smaller enrollments than 

^ttie 1973 .survey averages. The "eligible but missing" departments 1 
probably tend to be thfe smaller, newer, and less visible departments 
Experience gained in the next fev* years by using the, "universe of 
eligible departments" generate^ from catalogs will provide a more 
definitive assessment of the usefulness of catalogs for this 
purpose . 

DEPARTMENTAL RECORD^ CHECK ' ' : 1 ' 1 

The second ma j'or' component 6f the^ Quality Assessment 
Study was to reconstruct the 1973 'survey responses from 'departmental 
records' for a. sample of departments, and. compare the resulting 
data, with that which l^ad been reported by the department. The pro- 
cedure wAs 'designed to, provide data that could be used to assess 
the accuracy of .sutvey reporting* 
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^Field team members -visited the 30 institutions and 120 
departments "comprising the sample selected for the study, Fj.eld 
representatives compiled a listing of the graduate students and, 
. postdpctorates in^ each- department a^d then recorded the ^sajne 
-characteristic data used by the departments in compiling their 
s.urvey "responses. 



Comparison of the data obtained frbm the department 
records, by the field team' members with that ''reported by the "depart- - 
merits in the. 1973 survey revealed 'a difference of only 1.2 percent • 
in % the tc^tal number of graduate students reported by the two -sources. 
Also, the split between U.S. Government- and non-U. S. Government 
sources showed (Mose agreement, differing by only 0-8 an& 2,0 " 
percent, respectively, -Differences i,n type of support, including 
fellowships c ^nd traineeships , research and teaching assistkntships- 
and other types of support, ranged £rom two to nine percent, 
'In pene^al, the larger categori^es^of classification 1 were observed 
to have the greatest consistency "between departmental reports and 
field staff reports. -All data reciting from the quality - check'. 
% are subject to substantial sampling error, - * ■ * 

. The experience gained in_£&itipg the survey, questionnaires 
and conducting interviews with .department personnel revealed that \ 
hany respondents had difficulty completing their survey, responses^ 
This difficulty is. attributed) in part, to the complexity of th^v 
survey form >, which' required pretabulated student counts. Responses 
problems were ■ also, attributable . to the fact that a respondent ,did * 
riot always* have the required infc>rmation available to him. The ] 
Source of funds used for student support are not 'always k'nown at 
the department level, and ate frequently drawn from a pool of 
funds representing multiple sources/ Another problem is encountered 
when asking^for .student counts tabulated by "major? source of 
income,' in th$t department personnel mky be aware Of sources that 
are. Utilized by a given student if the funds are channeled through 
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the department. However, since they m$y not be aware of other 

sources used, such as student loans, family support, employment 

of spouse, G.I. Benefits savings , e'tc, as*king department person-' • 

nel to report by major source of income creates a problem. This 

is particularly true in the case of larger departments where an ^ 

individti^l student ar\d his/her resources may be less well known 

to department personnel. ' 

_ _ _ 9 

There was also evidence that some departments tend to* 
report a student as receiving a fellowship or assistantship , 
regardless of the amount of income the student derives from that 
particular source. ■ v 

STUDENT INTERVIEWS " 

* The student characteristics data obtained f*fom inter- 

views conducted with the sample of graduate students ^<ui post- 
doctorSls was compared with data extracted from department records. 
The interviews were designed t<£ obtain the same studer^^character- 
istics used by th«^3epartments in completing* their 1973 survey 
responses so that the accuracy of the- departments 1 records could 
be assessed, or their consistency with studdnt perceptions determined 

The comparisons showed net differences of ,less than three 
percent on sex, citizenship , yeax-in-program, apd type of support, • ' 
and approximately £i\/s, percent on souree-of -support (U.S. Govern- 
ment vs. non-U. S. 'Government) . Thus, most differences tend to 
"net out" in the comparison. Again, there is substantial sampling \ 
error in the comparisons. . - < 

A factor which" contributed # to the differences observed 
for type and source of support rs the fact that funds .used for 
student - support . frequently represent a mixture of both Federal and 
non-Federal. This can lea3 to mis-reporting by both the student 
and the department. : . ~ 



* The 1973 survey responses 'contained -student counts that 

were partially tabulated by type and source-of -support , and by 
year-in-progr^m and/or citizenship. It^ was 'not possible is© 
reconcile these data with the listings of student^ and their . 
characteristics as compiled by th$ field representatives.. Lack of 
the reconciliation step prohibits the identification of either data 
set as being the "correct"* set. For this reason, it is pdssible 
to estimate response inconsistencies, not response errors. 

\ 

^Since the sample for the R&V Study was small, dbkerved 
'differences, even ,though large, were usually v'ithin the expected 
range of sampling error. A positive outcome of the study, how- 
ever, is that observed inoonsistencies may be used to identify 
response categories and definitions that may bte troijbl^so^e. 

Positive steps have been ta^ken by the 1 agencies in the 
1974 survey to expand coverage and to simplify the questionnaire. 



